Augmented reality can be defined as adding or subtracting information to the human senses. Visual augmentations can be superimposed with a head-worn display Head-Worn Displays allow full mobility and private access to information. Head worn displays can be classified into two classes, optical and video see-through as described by Rolland and Fuchs, Optical versus video see-through head mounted displays in medical visualization, Presence Vol. 9, No. 3, Massachusetts Institute of Technology, 2002, pp. 287-309.
Some scientists prefer video see through, primarily for technical reasons. For example, using chroma keying it is easy to add information to the scene and it is relatively easy to block out parts of the scene. From a human usability point of view, there are several issues with video see-through augmented reality such as lag due to processing of the incoming video stream in video see through augmented reality is a problem.
For optical see-through displays, most optical designs, even today, combine the computer generated imagery with the real world using a beam splitter. In the early prior art, showing “opaque” objects with hidden lines removed was beyond capability and the three-dimensional objects were transparent wire frame line drawings.
It is well established that a beam-splitter will reflect some percentage of the incoming light and transmit the rest. The percentage of transmission and reflection can be adjusted through coatings on the surfaces of the beam-splitter. However, regardless of the transmission/reflection percentages, some light is always transmitted from the scene under various non-zero illumination settings, which is the source of the occlusion problem. This transmitted light implies that with only a beam-splitter and an image source based display, it is optically challenging to create mutual occlusion of real and computer-generated objects.
Alternative mechanisms to the beam-splitter plus image source combination are necessary to achieve the continuum between transparency and opaqueness of virtual objects, which is necessary to create various occlusion percepts. To be concrete about this transparency and opaqueness continuum of virtual objects, a virtual object needs to be opaque when it is occluding a real object, and it needs to be transparent when it is occluded by an object in the real world scene. Between these two extremes of the continuum, partial occlusion and associated transparency occur. Current computer graphics techniques and hardware allow for “hidden line removal” or visible surface determination, however, the display of computer-generated “opaque” objects with optical see-through remains a problem.
At a coarse scale, the illumination of the outside world can be controlled uniformly for the entire scene. U.S. Pat. No. 5,526,184 illustrates this idea of uniform illumination control of the scene by using a liquid crystal shutter. It is conceivable that electrochromic films can be used for the same purpose. On a finer scale, the scene can be thought to be composed of portions, and each portion can be individually modulated. U.S. Pat. No. 6,037,914 issued to Robinson on Mar. 14, 2000 discloses a system where a transmission type device was utilized to block/pass certain parts of the scene. Soon after, Eric W. Tatham, “Getting the best of both real and virtual worlds”, Communications of the ACM, Vol. 42, No. 9, September 1999, disclosed results from a transmissive light blocking arrangement with no imaging optics. An active mask was used to modulate the content of a scene and was combined with the display. Tatham further pointed to some potential benefits of using a Digital Micromirror Device (DMD) in place of the transmissive mask, yet no optical layout was proposed. Uchida, Sato and Inokuchi, “An optical See-through MR display with digital micro-mirror device, TVRSJ Vol. 7 No. 2, (2002), Abstract, shows a DMD based system in the proceedings of the Virtual Reality Society of Japan 2002. Uchida et.al's prototype benefits from the high contrast ratio of the DMD device, however, it is composed of three separate optical paths and further work is necessary to combine the paths for a head-mounted display application. However Kiyokawa et al., “An Occlusion-Capable Optical See-through Head Mount Display for Supporting Co-located Collaboration”, ISMAR 2003: pp. 133-141 addressed the occlusion problem in their prototype, ELMO-1, which is currently in its fourth generation. Vivid images of mutual occlusion were first demonstrated by Kiyokawa et al. using a system with a transmissive spatial light modulator (SLM). ELMO-4 optics is based on a 320×240 transmissive liquid crystal display produced by Hunet that is reported to have a response time of 2 ms. The ELMO-4 optical system contains four lenses, two prisms, and three mirrors per eye for the display component.
The system, method and apparatus of the present invention use a compact optical approach within an optical see-through display to performing occlusion, as opposed to using some form of video acquisition and graphics manipulation as done in video see-through displays. The benefit of the invention is that the system is compact and optical see-through because optical see-through displays are much faster than video see-through displays and they provide excellent resolution of the real scene because the scene is not sampled by cameras, instead, the human eye gets a direct view of the real world.
To overcome the problems and limitations of the prior art, the system, apparatus and method of the present invention is based on a reflective spatial light modulator, which has a resolution of 1280×1024, a response time within microseconds, and allow optimal compactness of the optical system. The increased response time is due to faster switching in ferroelectric liquid crystal within microseconds compared to numatic liquid crystals, which are an order of magnitude faster than the transmissive masks such as milliseconds or in best case scenario sub milliseconds. Per eye, the compactness is obtained using two lenses, either a single polarizing x-cube prism or a free standing wire grid polarizing structure, and a reflective spatial light modulator. The ferroelectric liquid crystal on silicon (F-LCOS) is a commercially available product from CRL Opto, United Kingdom. See Application Guide for CRL Opto SXGA FLCOS displays, CRL Opto Limited, United Kingdom (2003) [online]. [Retrieved on May 2, 2005]. Retrieved from: http://www.crlopto.products/product_support.htm.
In the prior art transmissive mask approach of ELMO-4, the liquid crystal display module reported a contrast ratio of 1:100. The F-LCOS, which is used as one implementation of the spatial light modulator panel in an embodiment of the present invention, is reported to yield a contrast ratio greater than approximately 1:200. F-LCOS also provides a light throughput between 40-50% of what the scene would provide for the real world objects. For the virtual objects, a light throughput of approximately 50% is achieved if the micro display used to paint the images is polarized such as liquid crystal display and ferroelectric liquid crystal on silicon panels, or approximately 100% for unpolarized displays such as organic light emitting diodes (OLEDs).